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Abstract 



Background/context: According to the National Assessment of Educational Progress (NAEP), 
just over 70 percent of students nationally arrive in high school with reading skills that are below 
“proficient” - defined as demonstrating competency over challenging subject matter. Nearly half 
of these students do not exhibit even partial mastery of the knowledge and skills that are 
fundamental to proficient work at grade level. 

Recent research indicates that struggling adolescent readers grapple with a constellation of reading 
difficulties that range from severe problems with basic literacy skills to troubles gaining a nuanced 
understanding of text. Biancarosa and Snow (2004) indicate that about “70 percent of older readers 
require some form of remediation.” 1 However, these students’ problem is less often with knowing 
how to read words on a page and rather more often with understanding what they read: they have 
difficulties with comprehension (Curtis and Chmelka, 1994). Their struggles with comprehension 
can stem from a lack of reading fluency, a lack of strategies for how to make sense of what they read, 
or a lack of experience employing such strategies across multiple types of texts (Schoenbach, 
Greenleaf, Cziko, and Hurwitz, 1999). 

These problems are especially acute as students navigate the transition into high school, facing new 
challenges that can easily push them off the path toward graduation and preparation for 
postsecondary education and work. Ninth-grade requirements represent a leap for entering freshmen, 
who face an increase in the amount of reading required in their courses, thicker and more 
intimidating textbooks, and new vocabulary that can be overwhelming. Struggling readers, who may 
have interest in academic subjects, but lack confidence in being able to improve their reading, may 
increasingly avoid challenging reading materials and try to avoid situations in which their poor 
reading skills will be exposed (Guthrie, 2002; Guthrie and Alvermann, 1999; Wigfield, 2004). 

Striving adolescent readers raise a critical challenge for high school reform initiatives that aim to 
improve low-performing high schools (Quint, 2006). Limitations in literacy skills are a major 
source of course failure, dropping out, and poor performance in postsecondary education, yet 
most high schools provide no formal instructional supports for literacy development. However, 
many English/language arts and social studies teachers do not see literacy development as their 
role (Shanahan 2004). Thus, secondary school instructional planning often reflects a belief that 
teaching reading is the domain of elementary schools (Roe, Stoodt and Burns, 1998). 

Purpose/objective/research question/focus of study: Unfortunately, little is known about school- 
based interventions that address the needs of struggling adolescent readers. To help fill these gaps in 
knowledge and to provide evidence-based guidance to practitioners, the U.S. Department of 
Education initiated the Enhanced Reading Opportunities (ERO) Study — a demonstration and 
random assignment evaluation of supplemental literacy programs targeted to ninth grade students 
with limited literacy s ki lls. 2 

The demonstration involves 34 high schools from 10 school districts that are implementing one 
of two supplemental literacy programs: Reading Apprenticeship Academic Literacy (RAAL), 
designed by WestEd, or Xtreme Reading, designed by the University of Kansas Center for 



'Biancarosa and Snow (2004) focus on students in grades 4 through 12. 

2 The ERO study is known more formally as “An Evaluation of the Impact of Supplemental Literacy Interventions in 
Freshman Academies.” 
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Research on Learning. The programs are supplemental as they consist of a year-long course that 
replaces a ninth-grade elective class and not a core academic class. They aim to help striving 
adolescent readers develop and apply the strategies and routines used by proficient readers and to 
motivate them to read more. The literacy programs were implemented in school years 2005-2006 
and 2006-2007, resulting in two cohorts of ninth-grade participants. The U.S. Department of 
Education’s Office of Elementary and Secondary Education (OESE) 3 provided direct support for 
implementation to the participating schools and districts, while the Institute of Education 
Sciences has been funding and overseeing the design and execution of the evaluation effort. 
MDRC, a nonprofit, nonpartisan social policy research organization, is conducting the evaluation 
in partnership with the American Institutes for Research and Survey Research Management. 

The study’s first report described the first year of implementation of the ERO programs and 
presented impact findings for the first cohort of ninth-grade students (2005-06). 4 The key impact 
finding was that overall, the ERO programs improved students’ reading comprehension test scores 
by 0.09 standard deviation (p-value = 0.019). 5 Although not statistically significant, the estimated 
impact of each literacy intervention (Xtreme Reading, RAAL) was also 0.09 standard deviation. 

This conference paper will present findings from the second report for the ERO study, which 
examined implementation and impacts for the second year of program operation. The following 
questions were examined for the second cohort of participating ninth-grade students: 

• What are the short-term impacts of the two supplemental literacy interventions, together and 
separately, on ninth-grade students’ reading s ki lls and behaviors? 

• For which subgroups of students are the ERO interventions most or least effective? 

• What factors promote or impede successful implementation of the ERO interventions? 

Setting: The supplemental literacy programs were implemented in 34 high schools located in 10 
school districts across the country. The districts were selected through a special grant competition 
organized by the U.S. Department of Education’s Office of Vocational and Adult Education 
(OVAE). Overall, ERO programs were implemented in schools located predominantly in large and 
midsize cities, with some of the schools in each of these categories being listed as “urban fringe.” As 
specified by the OVAE grant requirements, all schools enrolled more than 1,000 students in grades 9 
through 12, averaging 1,685 students total and 570 ninth-grade students per school. On average 
across the 34 high schools, the twelfth-grade class is 59 percent of the size of the ninth-grade class 
three years earlier, suggesting that roughly 41 percent of students have left the schools between the 
ninth and twelfth grades. This measure, “promoting power,” serves as a proxy for the likely 
longitudinal graduation rate. 6 Thirty-eight percent of the students in the participating schools were 
eligible for Title I services and that 47 percent of the students were approved for free or reduced- 
price lunch. 



3 The support originally came from the Office of Vocational and Adult Education (OVAE) and transferred later to 
OESE. 

4 See Kemple, Corrin, Nelson, Salinger, Drummond, and Herrmann (2008). 

5 Although they are not statistically significant, the magnitudes of the impact estimates for each literacy intervention 
were the same as those for the full study sample. 

6 Balfanz and Legters (2004) developed this measure of “promoting power” to approximate a school's graduation 
rate. It is calculated as the ratio of the number of twelfth-grade students in a given school year to the number of 
ninth-grade students from three years prior. 
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Population/Participants/Subjects: The target population for the study is comprised of ninth- 
grade students reading two to five years below grade level. In the second year of the study, the 
participating high schools identified 2,679 students whose pretest scores indicated that they were 
reading at this level (on average, 79 students per school). Prior to entering the study, students in 
the study sample were reading at a 4.9 grade level (an average of almost four years below grade 
level) and the 14th percentile nationally. On average, the study sample is over 78 percent 
Hispanic or black; 49 percent reported that a language other than English is spoken in their 
homes; and 29 percent are overage for grade (15 years old or older at the start of ninth grade, 
suggesting that they were retained in at some point in their schooling). 

Intervention/Program/Practice: The ERO study is a test of supplemental literacy interventions 
that are designed as full-year courses and targeted to ninth-grade students whose reading skills 
are two or more years below grade level as they enter high school. The two programs — Reading 
Apprenticeship Academic Literacy (RAAL) and Xtreme Reading — were selected for the study 
from a pool of 17 applicants by a national panel of experts on adolescent literacy. 

The programs were required to focus on instruction in the following areas: (1) student motivation 
and engagement; (2) reading fluency, or the ability to read quickly, accurately, and with 
appropriate expression; (3) vocabulary, or word knowledge; (4) comprehension, or making 
meaning from text; (5) phonics and phonemic awareness; and (6) writing. The overarching goals 
of both programs are to help ninth-grade students adopt the strategies and routines used by 
proficient readers, improve their comprehension skills, and be motivated to read more and to 
enjoy reading. Both programs are supplemental: they consist of a year-long course that replaces a 
ninth-grade elective class (rather than a core academic class), and they are offered in addition to 
students’ regular English language arts classes. 

Experienced, full-time English/language arts or social studies teachers volunteered and were 
approved by the U.S. Department of Education, the districts, and the schools to teach the 
programs for two years. One teacher per high school was trained and responsible to teach four 
sections of the ERO class exclusively. Each section accommodated 10 to 15 students. Classes 
were designed to meet for a minimum of 225 minutes per week. The developers for each ERO 
program provided training and technical assistance to ERO teachers: a three-day summer training 
institute, and off-site booster training sessions and on-site coaching visits during the school year. 

Research Design: The ERO evaluation utilizes a two-level random assignment research design. 
First, within each district, eligible high schools were randomly assigned prior to the first year of 
program implementation to use one of the two supplemental literacy programs: 17 high schools 
were assigned to use RAAL, and 17 schools were assigned to use Xtreme Reading. Each school 
implemented the same program in both the 2005-2006 and 2006-2007 school years. 

Second, eligible students within each of the high schools were randomly assigned to either enroll 
in the ERO class (ERO group) or to take one of their school’s regularly offered elective classes 
(non-ERO group). In the second year of the study, participating high schools identified 2,679 
ninth-grade students whose baseline test scores indicated that they were reading two to five years 
below grade level: 1,529 students (57 percent) randomly assigned to the ERO group and 1,150 
(43 percent) randomly assigned to the non-ERO group. There is no systematic difference 
between the background characteristics of the ERO students and the non-ERO students. 
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Data Collection and Analysis: The analysis examines impacts on students’ reading 
achievement and reading behaviors. Students’ reading achievement was measured using the 
Group Reading Assessment and Diagnostic Examination (GRADE) assessment. 7 The reading 
comprehension and vocabulary subtests of the GRADE were analyzed, with reading 

o 

comprehension as the study’s primary outcome of interest. The GRADE was administered prior 
to random assignment (“baseline”) and at the end of their ninth-grade year (“follow-up”). 

Information on students’ reading behaviors was collected using a follow-up student survey. 
Responses to the survey were used to derive measures of three reading behaviors: the number of 
times during the prior month that a student read different types of text in school or for homework; the 
number of times during the prior month that a student read different types of text outside of school; 
and students’ reported use of the reading strategies and techniques that the ERO programs try to 
teach. 

Information was also collected to learn about program implementation. To learn about program 
fidelity to developers’ specifications, the study included observations of the supplemental literacy 
classes during the first and second semester of the school year. Other implementation data collected 
included information about student course schedules and attendance in the ERO courses. 

The impact analysis is based on the 2,171 students who have follow-up GRADE test scores. The 
impact analysis uses ordinary least squares regression to estimate the difference in outcomes between 
the ERO and the non-ERO group, adjusted for the blocking of random assignment by school. To 
improve the precision of the impact estimates, the analysis also controls for random differences 
between the ERO and the non-ERO group in their GRADE reading comprehension score at baseline 
and whether they are overage for grade (an indicator of grade retention). 

Findings/Results: The key implementation findings from the second year of the study are that: 

• Implementation fidelity: The implementation fidelity of the ERO programs was more highly 
rated in the second year of the study than in the first year. In comparison to the first year, a 
greater number of schools in the second year were deemed to have programs that were well 
aligned with program developers’ specifications (26 schools in the second year compared to 
16 schools in the first year), and fewer schools were considered to be poorly aligned (one 
school in the second year compared to 10 schools in the first year). 

• Program start-up: Schools started offering their ERO classes sooner in the second year of 
implementation than in the first year (an average of 2.3 weeks after the start of the school 
year in the second year compared to 6 weeks in the first year). 

• Teachers’ experience with the program: Twenty-seven of the 34 ERO teachers from the first 
year of the study returned for the second year. 

Thus, in most of the study schools, the second cohort of participating students was exposed to a 
full year of program operation and to teachers in their second year of implementing the 
programs. Yet, despite the observed improvements in program implementation, the impact of the 
ERO programs in the second year of the study were similar to those found in the first year (see 
Table 1): 



American Guidance Service (2001a, 2001b). 

Tmpacts on both subtests are presented in standard score units; students with a standard score of 100 points are 
considered to be reading at grade level. 
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• In the second year of the study, the ERO programs produced a positive and statistically 
significant impact on students’ reading comprehension of 0.80 standard score points 
(which corresponds to an effect size of 0.08 standard deviation, p-value = 0.042). 9 

Figure 1 places this impact in the context of the actual and expected change in the ERO students’ 
reading comprehension test scores on the GRADE from the beginning of ninth grade to the end 
of ninth grade. The bottom section of the bar shows that at the start of their ninth-grade year, 
students in the ERO group achieved an average standard score of 84.6, corresponding to a grade 
equivalent of 4.9 (the last month of fourth grade) and an average reading level at the 14th 
percentile for ninth-grade students nationally. At the end of their ninth-grade year, the ERO 
group achieved an average standard score of 90.1, corresponding to a grade equivalent of 6.1 and 
an average reading level at the 25th percentile nationally. This means that the ERO group 
experienced a growth of 5.5 points in their reading comprehension skills in ninth grade. Had they 
not had the opportunity to attend the ERO classes, the ERO group would have grown 4.7 standard 
score points (as represented by the growth of the non-ERO group). Thus, the estimated impact of 
the ERO programs is 0.8 standard score points, which is a 17 percent improvement over and 
above the growth that the ERO group would have experienced without the ERO classes. 

Table 1 shows the estimated impact of each program. RAAL had a statistically significant, positive 
impact on reading comprehension test scores (effect size of 0.14 standard deviation). Xtreme 
Reading produced a positive impact (but not statistically significant) on the same measure (effect size 
of 0.02 standard deviation). The difference between these impacts is not statistically significant; thus, 
one cannot conclude that one program produced a different impact than the other. 10 

Conclusions: While the ERO programs improved students’ reading comprehension, the magnitude 
of their impact is modest. The ERO group’s reading comprehension scores still lagged 10 points 
below the national average at the end of ninth grade. In fact, 77 percent of students who participated 
in the ERO classes scored two or more years below grade level at the end of ninth grade, meaning 
that they would still be eligible for the ERO programs were the programs again available to them. 

It is important to remember, however, that the ERO interventions are modest in scope: they are “pull 
out” literacy classes for adolescents already far below grade level. Given the ERO study’s findings 
that a supplemental approach may not be enough, high schools may need to implement school -wide 
literacy reforms that embed literacy instruction in all content area classes (not just reading or EE A 
classes). School-wide reforms carry the potential for larger impacts; however, they are also more 
difficult to implement as they involve changing the instructional culture of a school. 

Finally, while the ERO programs have a modest impact on reading comprehension, they may have 
stronger impacts on students’ longer-term outcomes. The final report from this study will examine 
the impact of the programs on educational outcomes of both cohorts of students as they progress 
through high school (e.g., students’ performance in core classes and on high-stakes state assessments, 
their grade -to-grade promotion rates, and whether they are on track to graduate from high school). 



9 The overall impact of the programs in the second year (0.08 standard deviation) is not statistically different 
from the impact of the programs in the first year of implementation (0.09 standard deviation). 

10 The ERO programs - whether together or individually - did not have an impact on reading vocabulary or reading 
behaviors. Nor were impacts larger for some subgroups of students or schools than others. 
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Appendix B. Tables and Figures 



Table l.The Enhanced Reading Opportunities Study 
Impacts on Reading Achievement, Cohort 2 Follow-Up Respondent Sample 



Estimated , 

tor 







Non- 

ERO 


Estimated 


Impact 


Estimated 


Outcome 


ERO 

Group 


Group 


Impact 


Effect 

Size 


Impact 



All schools 



Reading comprehension 



Average standard score 


90.1 


89.3 


0.8 * 


0.08 * 


0.042 


Corresponding grade equivalent 


6.1 


6.0 








Corresponding percentile 


25 


23 








Reading vocabulary 












Average standard score 


93.5 


93.5 


0.0 


0.00 


0.986 


Corresponding grade equivalent 


7.8 


7.8 








Corresponding percentile 


32 


32 








Sample size 


1,264 


907 









Reading Apprenticeship schools 



Reading comprehension 



Average standard score 


90.2 


88.9 


1.4 * 


0.14 * 


0.015 


Corresponding grade equivalent 


6.1 


5.9 








Corresponding percentile 


25 


23 








Reading vocabulary 












Average standard score 


93.4 


93.8 


-0.4 


-0.04 


0.428 


Corresponding grade equivalent 


7.7 


7.8 








Corresponding percentile 


32 


33 








Sample size 


645 


470 









Xtreme Reading schools 



Reading comprehension 



Average standard score 


90.0 


89.7 


0.2 


0.02 


0.672 


Corresponding grade equivalent 


6.1 


6.0 








Corresponding percentile 


25 


24 








Reading vocabulary 












Average standard score 


93.5 


93.1 


0.4 


0.04 


0.468 


Corresponding grade equivalent 


7.8 


7.7 








Corresponding percentile 


32 


31 









2009 SREE Conference Abstract 

The Enhanced Reading Opportunities Study: Findings from the Second Year of Implementation 




Sample size 



619 



437 



Statistical significance is indicated (*) when the p-valne is less than or equal to .05. 

The difference in impacts between the two reading program is not statistically significant. 



Figure l.The Enhanced Reading Opportunities Study 
Impacts on Reading Comprehension, Cohort 2 Follow-Up Respondent Sample 




NOTES: The statistical significance is indicated (*) when the p-value is less than or equal to 5 
percent. The national average for standard score values is 100, and its standard deviation is 15. 
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